On Automated Evaluation of Readability of Summaries: Capturing Grammaticality, Focus, Structure and Coherence

Readability of a summary is usually graded manually on five aspects of readability: grammaticality, coherence and structure, focus, referential clarity and non-redundancy. In the context of automated metrics for evaluation of summary quality, content evaluations have been presented through the last decade and continue to evolve, however a careful examination of readability aspects of summary quality has not been as exhaustive. In this paper we explore alternative evaluation metrics for ‘grammaticality’ and ‘coherence and structure’ that are able to strongly correlate with manual ratings. Our results establish that our methods are able to perform pair-wise ranking of summaries based on grammaticality, as strongly as ROUGE is able to distinguish for content evaluations. We observed that none of the five aspects of readability are independent of each other, and hence by addressing the individual criterion of evaluation we aim to achieve automated appreciation of readability of summaries.